Moving image prediction encoding/decoding system

ABSTRACT

A moving image encoding/decoding system may include a video predictive encoding device, which may include: an encoding device which encodes each of a plurality of input pictures to generate compressed picture data including a random access picture, and encodes data about display order information of each picture; a restoration device which decodes the compressed picture data to restore a reproduced picture; a picture storage device which stores the reproduced picture as a reference picture; and a memory management device which controls the picture storage device. Following completion of an encoding process of generating the random access picture, the memory management device refreshes the picture storage device by setting every reference picture in the picture storage device, except for the random access picture, as unnecessary immediately before or immediately after encoding a picture with display order information larger than the display order information of the random access picture.

PRIORITY CLAIM

This application is a continuation of and claims the benefit of priorityfrom PCT Patent Application No. PCT/JP2011/055915, filed Mar. 14, 2011,which claims the benefit of priority of Japanese Patent Application No.2010-061337, filed Mar. 17, 2010, both of which are incorporated byreference.

TECHNICAL FIELD

The present invention relates to moving image predictionencoding/decoding system that may include either or both of a videopredictive encoding system and a video predictive decoding system.

BACKGROUND ART

Compression encoding technologies are used for efficient transmissionand storage of video data. The systems of MPEG1 to MPEG4 and H.261 toH.264 are widely used for videos. In these encoding systems, a pictureas an encoding target is divided into a plurality of blocks and eachblock is subjected to an encoding/decoding process. Predictive encodingmethods are used for enhancement of encoding efficiency.

SUMMARY

A moving image prediction encoding/decoding system includes an encodingsystem and a decoding system. The moving image predictionencoding/decoding system may achieve efficient compression encoding ofpictures before and after a picture at a random access point.

The encoding system may operate as a video predictive encoding devicethat includes input means, which accepts input of a plurality ofpictures constituting a video sequence; encoding means which encodeseach of the input pictures by a method of either intra-frame predictionor inter-frame prediction to generate compressed picture data includinga random access picture serving as a picture of random access, and whichencodes data about display order information of each of the pictures;restoration means which decodes the compressed picture data thusgenerated, to restore a reproduced picture; picture storage means whichstores the reproduced picture thus restored, as a reference picture tobe used for encoding of a subsequent picture; and memory managementmeans which controls the picture storage means, wherein followingcompletion of an encoding process to generate the random access picture,the memory management means refreshes the picture storage means bysetting every reference picture stored in the picture storage meansexcept for the random access picture as unnecessary immediately beforeor immediately after first encoding a picture with display orderinformation larger than the display order information of the randomaccess picture.

When encoding the display order information of the at least one encodingtarget that includes a picture which has display order informationlarger than the display order information of the random access pictureand becomes the first encoding target after completion of the encodingprocess of generating the random access picture, the encoding means mayencode a difference value between the display order information of atleast one encoding target and the display order information of therandom access picture.

When encoding the display order information of each picture in asequence from a picture that becomes the next encoding target after therandom access picture, to a picture having display order informationlarger than the display order information of the random access pictureand that becomes the first encoding target after completion of theencoding process of generating the random access picture, the encodingmeans may encode a difference value between the display orderinformation of each picture and the display order information of therandom access picture.

A video predictive decoding device according to an example embodiment isa video predictive decoding device comprising: input means which acceptsinput of compressed picture data including a random access pictureserving as a picture of random access, which was obtained by encodingeach of a plurality of pictures constituting a video sequence by amethod of either intra-frame prediction or inter-frame prediction, anddisplay order encoded data obtained by encoding data providing displayorder information of each of the pictures; restoration means whichdecodes the compressed picture data to restore a reproduced picture andwhich decodes the display order encoded data to restore the displayorder information thereof; picture storage means which stores thereproduced picture thus restored, as a reference picture to be used fordecoding of a subsequent picture; and memory management means whichcontrols the picture storage means, wherein after completion of adecoding process of decoding the random access picture the memorymanagement means refreshes the picture storage means by setting everyreference picture stored in the picture storage means except for thedecoded random access picture as unnecessary immediately before orimmediately after first decoding a picture having display orderinformation larger than the display order information of the randomaccess picture.

When decoding display order information of at least one decoding targetpicture which has display order information larger than the displayorder information of the random access picture and which becomes thefirst decoding target following completion of the decoding process ofdecoding the random access picture, the restoration means may restorethe display order information of the decoding target picture by adding adifference value to the display order information of the random accesspicture. The difference value may represent a difference between thedisplay order information of the decoding target picture and the displayorder information of the random access picture. The display orderinformation of the decoding target picture may be obtained by decodingthe display order encoded date of the decoding target picture.

When decoding display order information of each picture in a sequencefrom a picture which becomes a next decoding target after the randomaccess picture, to a picture having display order information largerthan the display order information of the random access picture, andwhich becomes the first decoding target after completion of a decodingprocess of generating the random access picture, the restoration meansmay restore the display order information of each picture by adding adifference value to the display order information of the random accesspicture. The difference value may represent a difference between thedisplay order information of each picture and the display orderinformation of the random access picture. The display order informationof each picture may be obtained by decoding the display order encodeddata of each picture.

A video predictive encoding method according to an example embodiment isa video predictive encoding method to be executed by a video predictiveencoding device with picture storage means for storing a referencepicture to be used for encoding of a subsequent picture, comprising: aninput step of accepting input of a plurality of pictures constituting avideo sequence; an encoding step of encoding each of the input picturesby a method of either intra-frame prediction or inter-frame predictionto generate compressed picture data including a random access pictureserving as a picture of random access, and encoding data about displayorder information of each of the pictures; a restoration step ofdecoding the compressed picture data thus generated, to restore areproduced picture; a picture storage step of storing the reproducedpicture thus restored, as a reference picture to be used for encoding ofa subsequent picture; and a memory management step of controlling thepicture storage means, wherein, following completion of an encodingprocess of generating the random access picture, in the memorymanagement step, the video predictive encoding device refreshes thepicture storage means by setting every reference picture stored in thepicture storage means, except for the random access picture, asunnecessary, immediately before or immediately after encoding a picturehaving display order information larger than the display orderinformation of the random access picture.

In the encoding step, the video predictive encoding device may encode adifference value. The difference value may be encoded as data providingdisplay order information of at least one encoding target picture. Theat least one encoding target picture may have display order informationlarger than the display order information of the random access pictureand may become the first encoding target picture following completion ofthe encoding process of generating the random access picture. Thedifference value may represent a difference between the display orderinformation of the encoding target picture and the display orderinformation of the random access picture.

In the encoding step, when encoding each picture in a sequence from apicture which becomes a next encoding target after the random accesspicture, to a picture which has display order information larger thanthe display order information of the random access picture, and becomingthe first encoding target after completion of the encoding process ofgenerating the random access picture, the video predictive encodingdevice may encode a difference value. The difference value may beencoded as data providing display order information of each picture. Thedifference value may represent a difference between the display orderinformation of each picture and the display order information of therandom access picture.

A video predictive decoding method according to an example embodiment isa video predictive decoding method to be executed by a video predictivedecoding device with picture storage means for storing a referencepicture to be used for decoding of a subsequent picture, comprising: aninput step of accepting input of compressed picture data including arandom access picture serving as a picture of random access, which wasobtained by encoding each of a plurality of pictures constituting avideo sequence by a method of either intra-frame prediction orinter-frame prediction, and display order encoded data obtained byencoding data about display order information of each of the pictures; arestoration step of decoding the compressed picture data to restore areproduced picture and decoding the display order encoded data torestore the display order information thereof; a picture storage step ofstoring the reproduced picture thus restored, as a reference picture tobe used for decoding of a subsequent picture, into the picture storagemeans; and a memory management step of controlling the picture storagemeans, wherein, after completion of a decoding process of decoding therandom access picture, in the memory management step, the videopredictive decoding device refreshes the picture storage means bysetting every reference picture stored in the picture storage meansexcept for the random access picture as unnecessary, immediately beforeor immediately after decoding a picture which has display orderinformation larger than the display order information of the randomaccess picture.

In the restoration step, for display order information of at least onedecoding target including a picture which has display order informationlarger than the display order information of the random access pictureand which becomes the first decoding target after completion of thedecoding process of decoding the random access picture, the videopredictive decoding device may restore the display order information ofthe decoding target picture by adding a difference value to the displayorder information of the random access picture. The difference value mayrepresent a difference between the display order information of thedecoding target picture and the display order information of the randomaccess picture. The display order information of the decoding targetpicture may be obtained by decoding the display order encoded data ofthe decoding target picture.

In the restoration step, when encoding the display order information ofeach picture in a sequence from a picture which becomes a next decodingtarget after the random access picture, to a picture having displayorder information larger than the display order information of therandom access picture and which becomes the first decoding target aftercompletion of a decoding process of generating the random accesspicture, the video predictive decoding device may restore the displayorder information of each picture by adding a difference value to thedisplay order information of the random access picture. The differencevalue may represent a difference between the display order informationof each picture and the display order information of the random accesspicture. The display order information of each picture may be obtainedby decoding the display order encoded data of each picture.

A video predictive encoding program according to an example embodimentis a video predictive encoding program for letting a computer operateas: input means which accepts input of a plurality of picturesconstituting a video sequence; encoding means which encodes each of theinput pictures by a method of either intra-frame prediction orinter-frame prediction to generate compressed picture data including arandom access picture serving as a picture of random access, and whichencodes data about display order information of each of the pictures;restoration means which decodes the compressed picture data thusgenerated, to restore a reproduced picture; picture storage means whichstores the reproduced picture thus restored, as a reference picture tobe used for encoding of a subsequent picture; and memory managementmeans which controls the picture storage means, wherein after completionof an encoding process of generating the random access picture, thememory management means refreshes the picture storage means by settingevery reference picture stored in the picture storage means except forthe random access picture as unnecessary, immediately before orimmediately after encoding a picture with display order informationlarger than the display order information of the random access picture.

A video predictive decoding program according to an example embodimentis a video predictive decoding program for letting a computer operateas: input means which accepts input of compressed picture data includinga random access picture serving as a picture of random access, which wasobtained by encoding each of a plurality of pictures constituting avideo sequence, by a method of either intra-frame prediction orinter-frame prediction, and display order encoded data obtained byencoding data about display order information of each of the pictures;restoration means which decodes the compressed picture data to restore areproduced picture and which decodes the display order encoded data torestore the display order information thereof; picture storage meanswhich stores the reproduced picture thus restored, as a referencepicture to be used for decoding of a subsequent picture; and memorymanagement means which controls the picture storage means, wherein aftercompletion of a decoding process of decoding the random access picture,the memory management means refreshes the picture storage means bysetting every reference picture stored in the picture storage meansexcept for the random access picture as unnecessary, immediately beforeor immediately after decoding a picture with display order informationlarger than the display order information of the random access picture.

The system as described above achieves efficient compression encoding ofpictures before and after a picture that is a random access point and,at the same time, resolves the inconveniences associated with thedefects of conventional technology.

The moving image encoding/decoding system uses the informationindicative of the display order attendant on each respective pictureforming a video sequence or compression-encoded picture data (which willbe referred to hereinafter as “display order information” (correspondingto the display time, temporal reference information, temporal reference,or the like, in the conventional technology)) to set the timing ofmemory refreshment. The memory refreshment may be carried out followingan intra-frame predicted picture (intra frame) at a random access pointto achieve efficient compression encoding of pictures before and afterthe random access picture in the display order and, at the same time,resolve the inconveniences associated with the defects of theconventional technology as described below.

Specifically, the display order information is attendant on each pictureand therefore there is no need for transmission of new information suchas a flag.

When a video sequence is edited (such as to discard some pictures, or tojoin other pictures), the display order information of each pictureforming the video sequence can be appropriately set, so as to cause nomalfunction.

Furthermore, the timing of memory refreshment by the moving imageencoding/decoding system is not limited to P pictures and is independentof the encoding types of pictures (I pictures, P pictures, or Bpictures), and therefore the processing can be performed in an encodingtype with the best encoding efficiency, independent of the necessity ofrefreshment of the memory.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a functional block diagram showing an example configuration ofa video predictive encoding device.

FIG. 2 is a functional block diagram showing an example configuration ofa video predictive decoding device.

FIG. 3 is an example operational flowchart showing a video predictiveencoding/decoding method according to an embodiment.

FIG. 4 is a schematic view illustrating an example of video predictiveencoding/decoding in accordance with the example operation illustratedin FIG. 3.

FIG. 5 is an example operational flowchart showing a video predictiveencoding/decoding method according to another example embodiment.

FIG. 6 is a schematic view illustrating an example of video predictiveencoding/decoding in accordance with the example operation illustratedin FIG. 5.

FIG. 7 is a drawing showing an example hardware configuration of acomputer for executing a program recorded in a recording medium.

FIG. 8 is a schematic view of an example of a computer for executing aprogram recorded in a recording medium.

FIG. 9 is a block diagram showing a configuration example of a videopredictive encoding system.

FIG. 10 is a block diagram showing a configuration example of a videopredictive decoding system.

FIG. 11 is a schematic view showing a prediction structure of an examplevideo predictive encoding/decoding method.

DESCRIPTION OF EMBODIMENTS

In intra-frame predictive encoding, a predicted signal is encoded. Thepredicted signal can be generated using a neighboringpreviously-reproduced image signal (restored image signal from imagedata previously encoded) in the same frame as a target block and adifference signal obtained by subtracting the predicted signal from asignal of the target block. In inter-frame predictive encoding, a searchfor a displacement signal is performed with reference to apreviously-reproduced image signal in a frame different from a targetblock. A predicted signal is generated with compensation for thedisplacement signal identified in the search, and a difference signalobtained by subtracting the predicted signal from the signal of thetarget block is encoded. The previously-reproduced image signal used asthe reference for the motion search and compensation can be called areference picture.

In bidirectional inter-frame prediction, reference can be made not onlyto past pictures that are to be displayed prior to a target picture inthe display time order, but also future pictures to be displayed afterthe target picture (provided that the future pictures need to be encodedprior to the target picture and are preliminarily reproduced). Then apredicted signal acquired from a past picture and a predicted signalacquired from a future picture can be averaged, which provides thebenefits of allowing effective prediction for a signal of anewly-appearing object, and reducing noise included in the two predictedsignals.

Furthermore, in inter-frame predictive encoding, such as using H.264, apredicted signal for a target block can be produced with reference to aplurality of reference pictures previously encoded and reproduced, and apicture signal with the smallest error can be selected as an optimumpredicted signal by motion search. Then a difference can be calculatedbetween a pixel signal of the target block and this optimum predictedsignal, and the difference can be subject to discrete cosine transform,quantization, and entropy encoding.

At the same time, a piece of information of a reference picture fromwhich the optimum predicted signal for the target block is acquired(reference index) and a piece of information of a region in thereference picture from which the optimum predicted signal is acquired(motion vector) can be encoded together. In inter-frame predictiveencoding, such as using H.264, four or five reproduced pictures can bestored as reference pictures in a frame memory. The frame memory, asdescribed herein, can include a reproduced picture buffer (decodedpicture buffer).

The inter-frame predictive encoding allows efficient compressionencoding by taking advantage of correlation between pictures, butdependence between frames is eliminated, in order to allow viewing fromanywhere in the middle of a video program, such as what can occur when aviewer is switching TV channels. A point without dependence betweenframes in a compressed bitstream of a video sequence will be referred tohereinafter as a “random access point.” Besides the switching of TVchannels, the random access points are also needed in cases of editing avideo sequence and joining compressed data of different video sequences.In inter-frame predictive encoding, such as using H.264, IDR picturescan be designated, the designated instantaneous decoding refresh (IDR)pictures can be encoded by the intra-frame predictive encoding method,and at the same time, reproduced pictures stored in the frame memory areset as unnecessary, so that the reproduced pictures are not used forreference pictures, thereby substantially clearing the frame memory (orrefreshing the frame memory). This process is hereinafter referred to as“memory refresh” and can also be called “frame memory refresh” or“buffer refresh” in some cases.

FIG. 11 (A) is a schematic diagram showing an example predictionstructure of a motion video including an IDR picture. A plurality ofpictures 901, 902, . . . , 909 shown in FIG. 11 (A) are part of a seriesof images constituting a video sequence. Each image is also called a“picture” or “frame.” Each arrow indicates a direction of prediction.For example, for the picture 902, a predicted signal is acquired usingpictures 903, 905 as reference pictures as indicated by the startingpoints of two arrows directed to the picture 902. The picture 901 inFIG. 11 (A) is assumed to be encoded with reference to past pictures notshown in FIG. 11 (A).

Next, the pictures 902, 903, and 904 are encoded using the bidirectionalpredictive encoding in order to increase compression rates.Specifically, the picture 905 is first encoded and reproduced and thenthe picture 903 is encoded with reference to the previously-reproducedpictures 901 and 905 (an arrow from the picture 901 is omitted in FIG.11 (A)). Thereafter, each of pictures 902 and 904 are encoded using thethree reproduced pictures 901, 905, and 903 as reference pictures (anarrow from the picture 901 is omitted in FIG. 11 (A)). Likewise,pictures 906, 907, and 908 are encoded with reference to pictures 905and 909.

The compressed data of the pictures that are encoded (or compressed) inthis manner is transmitted or stored in the order as described in FIG.11 (B). The correspondence or relationship between the compressed datain FIG. 11 (B) and the pictures in FIG. 11 (A) is indicated by commonidentifiers such as P1, IDR5, and B3. For example, compressed data 910is compressed data of picture 901 denoted by the same identifier “P1,”and compressed data 911 is compressed data of picture 905 denoted by thesame identifier “IDR5.”

Now, with consideration to random access, let us consider a case wherethe intra-frame predictive coding is carried out while designating thepicture 905 as an IDR picture. In this case, for example, immediatelyafter reproduction of the picture 905 by decoding of compressed data 911(or possibly immediately before the start of decoding of compressed data911), all the reference pictures stored in the frame memory (i.e., thepast reproduced pictures including the picture 901) are set asunnecessary so they are not used as reference pictures. As a result, thepicture 901 in FIG. 11 (A) is not available as a reference picture, andbecomes unavailable for encoding of pictures 902, 903, and 904.

Since the introduction of IDR pictures can lead to elimination ofreference pictures otherwise available for use in prediction, efficientencoding of pictures before an IDR picture is identified in the displayorder of pictures (the pictures 902, 903, and 904 in the example of FIG.11 (A)). To avoid this scenario, the timing of refreshment of the framememory (i.e., the timing of setting the reference pictures in the framememory as unnecessary) can be delayed until execution of encoding of apicture occurs, which can be encoded following designation of the IDRpicture. When the timing of refreshment of the frame memory is delayed,the picture 901 remains in the frame memory at the time of execution ofencoding the pictures 902, 903, and 904 in FIG. 11 (A), and thereforereference to the picture 901 is available when encoding the pictures902, 903, and 904, in order to allow efficient encoding.

The timing of the memory refreshment can be achieved by a number ofmethods, such as by a first method of adding information about thenumber of pictures to be delayed, to each IDR picture. In a secondexample method, compressed data can be added to each picture as asignal, such as a flag, that instructs execution of memory refreshment,where the signal corresponds to the timing of execution of memoryrefreshment. In a third example method, a P picture (unidirectionalpredicted picture) first appearing after each IDR picture can be definedas timing of refreshment.

However, the above methods have the following shortcomings. In method 1there is inconvenience created by editing of a video sequence, such aswhen some pictures out of a plurality of pictures are discarded andother pictures are joined or inserted, so as to make inappropriate the“information about the number of pictures to be delayed” that is addedto each IDR picture. In method 2, similar to the previous example, withuse of the flag, if compressed data of a corresponding picture isdeleted by editing of the video sequence, the flag added to the deletedcompressed data is missed. In method 3, since the sign (trigger) ofmemory refreshment is limited to the P pictures, encoding by othermethods becomes unavailable. For example, a picture at a change of scenecannot be encoded by intra-frame prediction (I picture). Due to theseshortcomings in the example methods, the timing of execution of memoryrefreshment can create a state in which there is no reference pictureavailable for decoding of subsequent data in the frame memory and, as aconsequence, reproduction of a subsequent picture can be challenging.

Video Predictive Encoding Device

FIG. 1 is a functional block diagram showing a configuration of a movingimage prediction encoding system in the form of a video predictiveencoding device 100 according to an example embodiment. As shown in FIG.1, the video predictive encoding device 100 is provided with functionalcomponents of input terminal unit 101, block divider unit 102, predictedsignal generator unit 103, frame memory unit 104, subtracter unit 105,transformer unit 106, quantizer unit 107, de-quantizer unit 108,inverse-transformer unit 109, adder unit 110, entropy encoder unit 111,output terminal unit 112, input terminal unit 113, and frame memorymanagement unit 114. Operation of the respective functional componentswill be described in the below-described operation of the videopredictive encoding device 100. The term “module” or “unit” or“component” is defined to include one or a plurality of executablemodules or units or components. As described herein, the modules orunits or components are defined to include software, hardware or somecombination thereof executable by a processor. Software modules mayinclude instructions stored in memory that are executable by theprocessor, Hardware modules may include various devices, components,circuits, gates, circuit boards, and the like that are executable,directed, and/or controlled for performance by the processor.

Example operation of the video predictive encoding device 100 will bedescribed below. A video signal of a video sequence consisting of aplurality of pictures as targets for an encoding process is fed into theinput terminal 101 and the block divider 102 divides each picture into aplurality of regions. In the present embodiment, each picture is dividedinto a plurality of blocks each consisting of 8×8 pixels, but it may bedivided into blocks of any block size or shape other than the foregoing.Next, for a target of a block as an object to be encoded (which will bereferred to hereinafter as “target block”), a predicted signal isgenerated by a below-described prediction method. In the presentembodiment, available prediction methods are two types of predictionmethods, inter-frame prediction and intra-frame prediction, and thebidirectional inter-frame prediction described in the background art isalso applicable to the inter-frame prediction. The respectivefundamental operations of the inter-frame prediction and the intra-frameprediction will be summarized below.

In inter-frame prediction, a reproduced picture having been previouslyencoded and then restored is used as a reference picture and motioninformation (e.g., a motion vector) is obtained from the referencepicture to provide a predicted signal with the smallest error for thetarget block. This process is called “motion detection.” In some cases,the target block may be subdivided into small regions and theinter-frame prediction method may be determined for a target of eachsubdivided small region. In such cases, the most efficient divisionmethod is determined among a variety of division methods. The determineddivision method is used to subdivide the target block into small regionsand motion information of each small region for the entire target blockare determined. In the present embodiment, the inter-frame prediction iscarried out by the predicted signal generator 103. The target block isfed through line L102 to the predicted signal generator 103, while thereference picture is fed through line L104 to the predicted signalgenerator 103. Concerning the reference picture, a plurality of pictureshaving been previously encoded and then restored can be used asreference pictures. Examples of uses of restored reference pictures areincluded in any one of the methods of MPEG-2, MPEG-4, and H.264, whichare conventional technologies. The determined division methodinformation used to determine the small regions, and motion informationof each small region is sent from the predicted signal generator 103through line L112 to the entropy encoder 111. The entropy encoder 111encodes the determined division method motion information and the motioninformation of each small region, and the encoded data is sent throughline L111 out of output terminal 112. Information indicating from whichreference picture the predicted signal is acquired out of the pluralityof reference pictures (reference index) is also sent from the predictedsignal generator 103 through line L112 to the entropy encoder 111. Thereference picture indication information is encoded by the entropyencoder 111, and then the encoded data is sent through line L111 out ofthe output terminal 112. In the present embodiment, as an example, fouror five reproduced pictures are stored in the frame memory 104 (orpicture storage medium), and are used as reference pictures. Thepredicted signal generator 103 acquires a reference picture from theframe memory 104, based on the small-region division method, and thereference picture and motion information for each small region, andgenerates a predicted signal from the reference picture and motioninformation (which is called “inter-frame predicted signal” in the sensethat it is a predicted signal obtained by inter-frame prediction). Theinter-frame predicted signal generated in this manner is sent throughline L103 to the subtracter 105 and to the adder 110 for below-describedprocessing.

On the other hand, the intra-frame prediction is to generate anintra-frame predicted signal, using previously-reproduced pixel valuesspatially adjacent to a target block. Specifically, the predicted signalgenerator 103 acquires previously-reproduced pixel signals in the sameframe from the frame memory 104 and generates a predicted signal byextrapolation of the previously-reproduced pixel signals (which iscalled “intra-frame predicted signal” in the sense that it is apredicted signal obtained by intra-frame prediction). The intra-framepredicted signal thus generated is sent from the predicted signalgenerator 103 through line L103 to the subtracter 105. The method ofgenerating the intra-frame predicted signal in the predicted signalgenerator 103 can be, for example, similar to the method of H.264. Theinformation indicating the extrapolation method in the intra-frameprediction is sent from the predicted signal generator 103 through lineL112 to the entropy encoder 111, where it is encoded by the entropyencoder 111, and the encoded data is sent out of the output terminal112.

The above summarized an example of respective operations of theinter-frame prediction and the intra-frame prediction. In practice, foreach target block, a predicted signal with the smallest error isselected from the inter-frame and intra-frame predicted signals obtainedas described above, and is sent from the predicted signal generator 103through line L103 to the subtracter 105.

Incidentally, since there is no previous picture for the first pictureto be encoded, all the target blocks in the first picture are processedby the intra-frame prediction. In preparation for switching of TVchannels, all target blocks in a certain picture are periodicallyprocessed as a random access point, by the intra-frame prediction. Suchpictures can be called intra frames.

The subtracter 105 subtracts the predicted signal received through lineL103, from the signal of the target block received through line L102, togenerate a residual signal. This residual signal is transformed bydiscrete cosine transform by the transformer 106 and each of thetransform coefficients are quantized by the quantizer 107. Finally, thequantized transform coefficients are encoded by the entropy encoder 111and the resultant encoded data is sent along with the information aboutthe prediction method through line L111 out of the output terminal 112.

On the other hand, for the intra-frame prediction or the inter-frameprediction for a subsequent target block, the quantized transformcoefficients (encoded data of the target block) are de-quantized by thede-quantizer 108 and thereafter the transform coefficients are inverselytransformed by inverse discrete cosine transform by theinverse-transformer 109, thereby restoring the residual signal. Then theadder 110 adds the restored residual signal to the predicted signal sentthrough the line L103, to reproduce the signal of the target block, andthe reproduced signal thus obtained is stored into the frame memory 104.The present embodiment employs the transformer 106 and theinverse-transformer 109, but any other transform process may be employedinstead of these. Furthermore, the transformer 106 and theinverse-transformer 109 may be omitted in some cases.

Incidentally, the capacity of the frame memory 104 is limited and it isactually impossible to store all reproduced pictures. For this reason,only reproduced pictures used for encoding of a subsequent picture arestored in the frame memory 104. A unit to control the frame memory 104is the frame memory management unit 114. The frame memory managementunit 114 controls the frame memory 104 in such a manner that the oldestreproduced picture is deleted out of N (e.g., N=4) reproduced picturesstored in the frame memory 104, to allow the most recent reproducedpicture used as a reference picture, to be stored in the frame memory104. In fact, the frame memory management unit 114 receives input ofdisplay order information of each picture and type information forencoding of each picture (intra-frame predictive encoding, inter-framepredictive encoding, or bidirectional predictive encoding) from theinput terminal 113, and the frame memory management unit 114 operatesbased on these pieces of information. At this time, the display orderinformation of each picture is sent from the frame memory managementunit 114 through line L114 to the entropy encoder 111, where it isencoded by the entropy encoder 111. The display order information thusencoded is sent together with the encoded picture data through line L111out of the output terminal 112. The display order information isinformation that is attendant on each picture, and may be informationindicative of an order of the picture, or information indicative of atime of display of the picture (e.g., a display reference time of thepicture (temporal reference)). In the present embodiment, for example,the display order information itself is encoded by binary encoding. Thecontrol method by the frame memory management unit 114 will be describedlater.

Video Predictive Decoding Device

Next, the video predictive decoding system, such as a video predictivedecoding device will be described. FIG. 2 is a functional block diagramshowing an example configuration of video predictive decoding device 200according to an example embodiment. As shown in FIG. 2, the videopredictive decoding device 200 is provided with functional components ofinput terminal 201, data analyzer unit 202, de-quantizer unit 203,inverse-transformer unit 204, adder unit 205, predicted signal generatorunit 208, frame memory unit 207, output terminal unit 206, and framememory management unit 209. Operations of the respective functionalcomponent will be described in operation of the video predictivedecoding device 200 described below. The means associated with decodingdoes not always have to be limited to the de-quantizer 203 andinverse-transformer 204. In other embodiments, any means other thanthese may be employed. In some example embodiments, the means associatedwith decoding may be composed of only the de-quantizer 203, without theinverse-transformer 204.

The operation of the video predictive decoding device 200 will bedescribed below. The compressed data obtained by the aforementionedencoding method is fed through the input terminal 201. This compresseddata contains the residual signal of the target block, the predictionsignal generation information describing generation of the predictedsignal, the quantization parameter, the display order information of thepicture, and the encoding type information indicating the encoding typeof the picture. Among these, the prediction signal generationinformation, for example in the case of the inter-frame prediction,contains the information about block division (the small-region divisionmethod information (e.g., the size of block or the like)), the motioninformation of each small region, and the reference index. In the caseof the intra-frame prediction, the prediction signal generationinformation contains the information about the extrapolation method.

The data analyzer 202 extracts the residual signal of the target block,the prediction signal generation information associated with thegeneration of the predicted signal, the quantization parameter, thedisplay order information of the picture, and the encoding typeinformation indicating the encoding type of the picture from the inputcompressed data. Among these, the residual signal of the target blockand the quantization parameter are fed through line L202 to thede-quantizer 203, the de-quantizer 203 de-quantizes the residual signalof the target block on the basis of the quantization parameter, and theinverse-transformer 204 inversely transforms the result of thede-quantization by inverse discrete cosine transform. The residualsignal restored in this manner is sent through line L204 to the adder205.

On the other hand, the extracted prediction signal generationinformation describing the generation of the predicted signal is sentthrough line L206 b to the predicted signal generator 208. The predictedsignal generator 208 acquires an appropriate reference picture out of aplurality of reference pictures stored in the frame memory 207 (orpicture storage medium), based on the prediction signal generationinformation describing the generation of the predicted signal, andgenerates a predicted signal on the basis of the appropriate referencepicture. The predicted signal thus generated is sent through line L208to the adder 205, and the adder 205 adds the predicted signal to therestored residual signal, so as to reproduce the signal of the targetblock. The signal of the target block thus reproduced is output throughline L205 from the output terminal 206 and, at the same time, it isstored as a reproduced picture into the frame memory 207.

Reproduced pictures used for decoding or reproduction of a subsequentpicture are stored in the frame memory 207. The frame memory managementunit 209 controls the frame memory 207 in such a manner that the oldestreproduced picture is deleted out of N (which is N=4 as an exampleherein, but may be any predetermined integer). The oldest reproducedpicture stored in the frame memory 207 is deleted to allow the mostrecent reproduced picture used as a reference picture, to be stored intothe frame memory 207. The frame memory management unit 209 operatesbased on the display order information of the target picture and theinformation about the encoding type of the picture, which are fedthrough line L206 a. The control method by the frame memory managementunit 209 will be described later.

An intra frame (intra-frame predicted picture) can serve as a randomaccess point. An IDR picture (instantaneous decoder refresh), such asthose included in H.264, may also be referred to as an intra-framepredicted picture. This name originates from the fact that the framememory (decoder buffer) is refreshed instantaneously after encoding ordecoding of an IDR picture. In contrast, the presently describedembodiments execute refreshment of the frame memory after a temporarystandby (or delay), instead of executing the refreshment of the framememory immediately after encoding or decoding of an intra frame as arandom access point (or immediately before the encoding or thedecoding). Thus, in the presently described embodiments this picture iscalled a DDR picture (deferred decoder refresh or delayed decoderrefresh). As described below in detail, the timing of refreshment of theframe memory is determined based on comparison between the display orderinformation of a DDR picture and the display order information of apicture as a target for processing (encoding or decoding) (which will bereferred to hereinafter as “processing target picture”).

Characteristic Processing Operations of Video Predictive Encoding Methodand Video Predictive Decoding Method

The operations of the video predictive encoding method and the videopredictive decoding method according to the moving image predictionencoding/decoding system will be described below with reference to FIGS.3 and 4. FIG. 3 is a flowchart showing example operation of the videopredictive encoding/decoding method according to the present embodiment.FIG. 3 will be described below as the video encoding method. However,FIG. 3 is also applicable to the video decoding method.

First, meanings of variables used in FIG. 3 will be described. TR meansdisplay order information, TR_DDR means display order information of aDDR picture, TR_CUR means display order information of a processingtarget picture at a point of interest or at a time of processing theprocessing target picture such that the processing target picture is thecurrent target picture, and RP means a state variable indicative ofwhether refreshment of the frame memory 104 is in standby. A case ofRP=1 indicates a state in which after a DDR picture becomes a processingtarget, refreshment of the frame memory 104 has not yet been executed(i.e., a state in which refreshment of the frame memory is in standby),and a case of RP=0 indicates a state in which refreshment of the framememory 104 has already been executed, or a state in which therefreshment process is not needed.

In FIG. 3, at a start of encoding of a video signal, first, TR_DDR andRP are initialized to 0 (step 301). Step 302 is to check whether RP=1and whether TR_CUR of the processing target picture is larger thanTR_DDR of the DDR picture. When these conditions are met, it isindicated that the frame memory refreshment is in standby and that theprocessing target picture is a picture in the series of pictures after aDDR picture, and thus the refreshing process of the frame memory 104(i.e., a process of setting reference pictures stored in the framememory 104, as unnecessary) is executed (step 303). It is, however,noted that the reference pictures stored in the frame memory 107 thatare set as unnecessary are only reference pictures with the displayorder information TR smaller than the display order information of themost recent DDR picture (TR_DDR). The most recent DDR picture (orintra-frame predictive encoded picture) stored in the frame memory 104is not set to be unnecessary. After completion of the refreshing processas described above, the state variable RP is set to RP=0.

On the other hand, when the aforementioned conditions are not met instep 302, the operation proceeds to step 304 to check whether thecurrent processing target picture is a DDR picture. It is assumed in thevideo predictive encoding device 100 that the encoding type informationabout the encoding type of the picture (DDR, inter-frame predictiveencoding, or bidirectional predictive encoding) is supplied through theinput terminal 113 in FIG. 1 from a control device (not shown). When itis determined in step 304 that the current processing target picture isa DDR picture, step 305 is carried out to set the display orderinformation TR_CUR of the current processing target picture to TR_DDRand to set the state variable RP to RP=1, and then the operationproceeds to step 306. On the other hand, when the condition is notsatisfied in step 304, the operation proceeds to step 306.

Step 306 is to obtain a reproduced picture corresponding to theprocessing target picture. In this step, the processing target pictureis encoded to obtain compressed data that is compressed by the encodingmethod described with reference to FIG. 1, and the compressed data isfurther decoded to obtain a reproduced picture (the reproduced picturecorresponding to the processing target picture). The compressed dataobtained by encoding is sent to the outside of the video predictiveencoding device 100. Alternatively, the compressed data may be stored ina memory (not shown) that may be included in the video predictiveencoding device 100. Next step 307 is to determine whether thereproduced picture corresponding to the processing target picture is tobe used as a reference picture in a subsequent process. Thisdetermination is made based on the encoding type of the picture. It isassumed in the present embodiment that a DDR picture, a unidirectionalpredictive encoded picture, and a specific bidirectional predictiveencoded picture all are determined to be used as reference pictures,which are stored. It is, however, noted that the present embodiments arenot limited to these encoding types or determination method.

When it is determined in step 307 that the reproduced picture is notused as a reference picture, the reproduced picture is not stored in theframe memory 104 and the operation proceeds to step 309. On the otherhand, if it is determined in step 307 that the reproduced picture isused as a reference picture, step 308 is carried out to store thereproduced picture in the frame memory 104, and then the operationproceeds to step 309.

At step 309 it is determined whether there is a next picture(unprocessed picture), and if there is a next picture, the operationreturns to step 302 to repeat the processes of steps 302 to 308 for thenext picture. The processes of steps 302 to 308 are repeatedly carriedout until the last picture is processed. In this manner and, aftercompletion of the processing for all the pictures, the processing ofFIG. 3 is terminated.

By the above-described processing of FIG. 3, after completion of theprocessing of a random access picture (the most recent DDR pictureherein), the frame memory 104 is refreshed at a time of processing apicture having display order information (TR) larger than TR_DDR (infact, in step 303 before the process of step 306). The timing ofrefreshing the frame memory may be at any time after completion of theprocessing of the random access picture (the most recent DDR pictureherein) when processing a picture with the display order information TRlarger than TR_DDR, and may occur immediately after the process of step306.

The aforementioned processing of FIG. 3 corresponds to the overallprocessing of the video predictive encoding device 100 in FIG. 1, and,particularly, the processes of steps 302 to 305 are carried out by theframe memory management unit 114.

FIG. 3 was described as the video encoding method, but is alsoapplicable to the processing of the video decoding method. In executionof the decoding processing, step 301 further includes receipt of data ofa compression-encoded picture (bitstream). The display order informationand encoding type of a target picture are extracted from the data andthe operations of steps 302 to 305 are carried out by the same method asabove. In execution of the decoding process, step 306 carries out aprocess of decoding the compressed data of the target picture to restorethe picture. The processes of step 307 and the subsequent steps are asdescribed above. This processing corresponds to the overall processingof the video predictive decoding device 200 in FIG. 2 and, particularly,the processes of steps 302 to 305 are carried out by the frame memorymanagement unit 209.

FIG. 4 is a schematic diagram for explaining example processing of thevideo predictive encoding/decoding method according to the presentembodiment. Pictures 401 to 409 shown in FIG. 4 are some of a series ofpictures constituting a video sequence, and the picture 401 indicates astate in which there are n pictures prior thereto. Therefore, as shownin region 418 of FIG. 4, the display order information TR of picture 401is represented by (n+1). Since the present embodiment is assumed toperform the encoding/decoding processing including bidirectionalprediction, FIG. 4 shows a state in which the picture 402 with TR=(n+5)is first processed, and thereafter the pictures 403, 404, and 405, whichare supposed to be displayed prior to the picture 402 are processed. Forthe same reason, the picture 403 with the display order of (n+3) isprocessed prior to the picture 404 with the display order of (n+2). Thisorder is the same as in FIG. 11 (B). It is noted that “process apicture” hereinafter refers to “encode or decode a picture.”

The identifiers written in frames of pictures 401 to 409 in FIG. 4 havethe following meanings. Namely, “P” means a picture encoded byunidirectional prediction, “DDR” means a picture encoded as a DDRpicture, and each of “B” and “b” means a picture encoded bybidirectional prediction. The pictures except for those indicated byuncapitalized b (i.e., pictures indicated by capitalized B, P, and DDR)all are assumed to be used as reference pictures. The value of RP foreach picture in region 420 and the value of TR_DDR in region 419 in FIG.4 are values immediately after completion of processing for eachpicture, but are not values at a start of processing for each picture(i.e., at the time of entry into step 302 in FIG. 3). For example, RP=0at the start of the processing for the picture 402, but RP=1 immediatelyafter completion of the processing for the picture 402.

In the processing of the picture 401, since the picture 401 is not a DDRpicture, it results in RP=0. TR_DDR corresponding to the picture 401 maytake any value, except a value stored by the preceding processing isset. Since the picture 401 indicated by capitalized P1 is used as areference picture, it is stored into the frame memory.

Subsequently, the processing of the picture 402 will be described withreference to FIG. 3. At this time, the reproduced picture P1 is storedin the frame memory, as shown in region 410 in the bottom row in FIG. 4.Since RP=0 at the time of the start of processing of the picture 402,step 302 results in negative determination and the operation proceeds tostep 304. Since the picture 402 is a DDR picture, step 304 results inpositive determination and step 305 is carried out to set RP=1 andTR_DDR=n+5. Since the picture 402 is used as a reference picture, it isstored into the frame memory.

At a point of starting processing of the next picture 403, as shown inregion 411 in FIG. 4, the pictures P1 and DDR5 are stored in the framememory. At this time, RP=1, but the display order TR (n+3) of thepicture 403 is smaller than TR_DDR (n+5) and the picture 403 is not aDDR picture; therefore, steps 302, 304 result in a negativedetermination and the picture 403 is encoded or decoded as it is (step306). Since the picture 403 is used as a reference picture, it is storedin the frame memory.

On the occasion of processing the pictures 404 and 405, refreshment ofthe frame memory is still in a standby state (RP=1). Since the pictures404 and 405 are not used as reference pictures, the pictures 404 and 405are not stored into the frame memory as shown in regions 412, 413 inFIG. 4, while the pictures P1, DDR5, and B3 remain stored therein.

RP=1 at a point of a start of processing of the picture 406; since thedisplay order information TR (n+9) of the picture 406 is larger thanTR_DDR (n+5), step 302 results in positive determination and step 303 iscarried out to set the reference pictures as unnecessary, to refresh theframe memory, and set RP=0. The reference pictures set as unnecessary atthis time are only the reference pictures with the display orderinformation TR smaller than that of the most recent DDR picture 402,except for the most recent DDR picture 402. Therefore, as shown inregion 414 in FIG. 4, storage areas of the picture P1 and the picture B3are released in the frame memory, with the result that only the pictureDDR5 remains stored. The picture 406, which is used as a referencepicture, is stored into the frame memory after completion of theprocessing of the picture 406, as shown in region 415 in FIG. 4, andthereafter the refresh control of the frame memory is carried out in thesame manner as above.

Since the reference picture in the frame memory (picture P1 in FIG. 4)is not set as unnecessary, immediately after or immediately before theprocessing of the DDR picture 402 as described above, reference can bemade to the picture P1 in the processing of the pictures 403, 404, and405 processed after the DDR picture 402, and this contributes to animprovement in encoding efficiency. Since the most recent DDR picture402 (picture DDR5) is not set as unnecessary in execution of refreshmentof the frame memory after the processing of the DDR picture 402, themost recent DDR picture 402 (picture DDR5) can be used as a referencepicture in the processing of the subsequent pictures 407, 408, and 409.

As described above, the present embodiment makes use of the displayorder information included with each respective picture to set thetiming of the memory refreshment that is carried out after theprocessing of the intra-frame predicted picture (DDR picture) serving asa point of random access. The timing of the memory refreshment is basedon the display order information, thereby achieving efficientcompression encoding of pictures before and after a random accesspicture. It also resolves the inconveniences associated with the defectsof the conventional technology, as described below.

Namely, since the display order information is always includes with eachrespective picture, there is no need for transmission of new information(flag), which resolves the defect 2 of the conventional technology.Furthermore, in the case of editing of a video signal (e.g., to discardsome of pictures or to join different pictures), pieces of display orderinformation of the respective pictures constituting the video signal arealso appropriately set so as to cause no malfunction, which resolves thedefect 1 of the conventional technology. Furthermore, since the timingof the memory refreshment according to the present invention is notlimited to P pictures, and is independent of the encoding types ofpictures (I pictures, P pictures, and B pictures), each picture isprocessed in an encoding type with the highest encoding efficiency,independent of the necessity of refreshment of the memory, whichresolves the defect 3 of the conventional technology.

Another Embodiment

The foregoing embodiments describe the processing in the case where thedisplay order information of each picture was encoded as an “absolutevalue.” In another embodiment, the display order information of eachpicture is encoded as “difference value,” in order to increase theencoding efficiency. The below will describe the embodiment in which thedisplay order information is encoded as “difference value.”

FIG. 5 shows a flowchart of an operational example of another embodimentof the video predictive encoding/decoding method. In this embodiment thedisplay order information of each picture is encoded as follows. Namely,for each picture that becomes a processing target during standby ofrefreshment of the frame memory (i.e. RP=1), a difference value betweenthe display order information of the target picture and the displayorder information of the DDR picture is encoded. On the other hand, foreach picture that becomes a processing target at a time when refreshmentof the frame memory 104 has already been executed, or at a time when therefreshment process is not needed (i.e. RP=0), the display orderinformation thereof is encoded by any method. For example, a differencefrom the display order information of the DDR picture may be encoded, ora difference from the display order information of an immediatelypreceding picture in the encoding order may be encoded.

In the example operation of this embodiment, FIG. 5 will be described asthe video decoding method, but it should be understood that FIG. 5 isalso applicable to the video encoding method. Step 501 in FIG. 5 is toreceive input data of a compression-encoded picture into the videopredictive decoding device 200, and to extract from the data, adifference value (delta_TR) of the display order information of thetarget picture, and information about the encoding type of the picture.At the same time, TR_DDR and RP are initialized to 0.

Next step 502 is to check whether RP=1. When this condition is met, itis meant thereby that the refreshment of the frame memory is on standby,and thus the operation proceeds to step 503. Step 503 is to set thedisplay order information TR_CUR of the current processing targetpicture to the sum of TR_DDR and delta_TR.

Next, step 504 is to check whether TR_CUR is larger than TR_DDR. Whenthis condition is met, it means that the refreshment of the frame memoryis on standby (RP=1) and that the processing target picture is a pictureafter the DDR picture in the display order, and thus the refresh processof the frame memory 207 (i.e., a process of setting the referencepictures stored in the frame memory 207, as unnecessary) is executed(step 505). However, the reference pictures set as unnecessary are onlythe reference pictures with the display order information TR smallerthan the display order information of the most recent DDR picture(TR_DDR). The most recent DDR picture (or intra-frame predictive encodedpicture) is not set as unnecessary. After completion of the refreshprocess as described above, the state variable RP is set to RP=0.Thereafter, the operation proceeds to below-described step 507. When theaforementioned step 504 results in negative determination, the operationalso proceeds to step 507.

On the other hand, when step 502 results in a negative determination(i.e. RP=0), the operation proceeds to step 506 to set TR_CUR to the sumof the display order information TR_PREV of a previously processedpicture and delta_TR, and then the operation proceeds to step 507.

Step 507 is to check whether the current processing target picture is aDDR picture. The video predictive decoding device 200 can obtain theencoding type information about the encoding type of the picture (DDR,inter-frame predictive encoding, or bidirectional predictive encoding)from the compression-encoded data input from the outside.

When it is determined in step 507 that the current processing targetpicture is a DDR picture, step 508 is carried out to set the displayorder information TR_CUR of the current processing target picture toTR_DDR and set the state variable RP to RP=1, and then the operationproceeds to step 509. On the other hand, when the condition is not metin step 507, the operation proceeds to step 509.

Step 509 is to obtain a reproduced picture corresponding to theprocessing target picture. In this case, the reproduced picturecorresponding to the processing target picture is obtained by decodingthe compressed data of the processing target picture by the decodingmethod described with reference to FIG. 2. The reproduced pictureobtained herein is sent, for example, external to the video predictivedecoding device 200. Next step 510 is to determine whether thereproduced picture corresponding to the processing target picture is tobe used as a reference picture in subsequent processing. Thisdetermination is made based on the encoding type of the picture. In thiscase, a DDR picture, a unidirectional predictive encoded picture, and aspecific bidirectional predictive encoded picture all are determined tobe reference pictures. It is, however, noted that the present inventionis not limited to these encoding types or determination method.

When it is determined in step 510 that the reproduced picture is notused as a reference picture, the operation proceeds to step 512 withoutstoring the reproduced picture into the frame memory 207. On the otherhand, when it is determined in step 510 that the reproduced picture isused as a reference picture, step 511 is carried out to store thereproduced picture into the frame memory 207, and then the flow proceedsto step 512.

Step 512 is to set TR_CUR to TR_PREV, for the subsequent process of step506, and then the operation proceeds to step 513. Step 513 is todetermine whether there is a next picture (unprocessed picture), and ifthere is a next picture, the operation returns to step 502 to repeat theprocesses of steps 502 to 512 for the next picture. The processes ofsteps 502 to 512 are repeatedly carried out up to the last picture inthis manner and after completion of the processing for all the pictures,the processing of FIG. 5 is terminated.

By the above-described processing operation of FIG. 5, after completionof the processing of a random access picture (the most recent DDRpicture) the frame memory is refreshed at a time when a picture havingdisplay order information TR that is larger than TR_DDR is processed (infact, in step 505 before the process of step 509). The timing ofrefreshment of the frame memory may be any time after completion of theprocessing of the random access picture (the most recent DDR pictureherein), when processing a picture with display order information TRthat is larger than TR_DDR, and may be a time immediately after theprocess of step 509.

The aforementioned processing of FIG. 5 corresponds to the overallprocessing of the video predictive decoding device 200 in FIG. 2 and,particularly, steps 502 to 508 are carried out by the frame memorymanagement unit 209.

The operation of FIG. 5 was described as a video decoding method but itis also applicable to the processing of a video encoding method. In thecase of execution of encoding processing, step 503 is to obtain delta_TRfrom the difference between TR_CUR and TR_DDR, and step 506 is todetermine delta_TR from the difference between TR_CUR and TR_PREV,followed by entropy encoding. Furthermore, step 509 is to encode thetarget picture and then decode the picture. This processing correspondsto the overall processing of the video predictive encoding device 100 inFIG. 1 and, particularly, the processes of steps 502 to 508 are carriedout by the frame memory management unit 114.

FIG. 6 is a schematic diagram for explaining the processing of the videopredictive encoding/decoding method according to the example embodimentof FIG. 5. Pictures 601 to 609 shown in FIG. 6 are some of a series ofpictures constituting a video sequence and show the same processing asthe pictures 401 to 409 described with reference to FIG. 4. However,FIG. 6 includes delta_TR shown in region 621, in addition to the regionsof FIG. 4. As seen from region 621, determination of delta_TR isdifferent depending upon the value of RP at a start of the encodingprocess of a target picture (the RP value of a previous picture).Namely, in the encoding processes of pictures 603 to 606, delta_TR isobtained as a difference value between TR of each picture and TR_DDR. Inthe encoding processes of picture 607 and the subsequent pictures,delta_TR is obtained as a difference value between TR of a targetpicture and TR of a picture immediately before the target picture. Forexample, TR of picture 607 is subtracted from TR of picture 606 toobtain delta_TR of picture 607. On the other hand, when the displayorder information TR is restored from the difference value delta_TR inthe decoding process of each picture, the display order information TRis restored by adding the difference value delta_TR obtained by decodingthe compressed data of the difference value, to TR_DDR. The processingthereafter is the same as that in FIG. 4 and is thus omitted herein.

In FIG. 6, even if the pictures 603 to 605 are missed by editing, sincethe display order information TR of the picture 606 is determined fromTR_DDR, it can be correctly reconstructed asTR=delta_TR+TR_DDR=4+(n+5)=n+9, and the refreshment of the frame memorycan be controlled without malfunction. If delta_TR of every picture isobtained as a difference value between the display order information ofthe picture and the display order information of a picture immediatelybefore it in the decoding order, and if the picture 603 is missed, thedisplay order information cannot be correctly reproduced and refreshmentof the frame memory will be executed at the timing of the picture 605(though, originally, the timing of the picture 606 is correct timing).

In the case where the embodiment of FIG. 6 is applied to the videoencoding process, when encoding the display order information of eachpicture (pictures 603-606) and awaiting refreshment of the frame memory,after completion of the processing of the random access picture (themost recent DDR picture herein), the difference value delta_TR betweenthe display order information TR of the current picture and the displayorder information TR_DDR of the DDR picture may be encoded, instead ofencoding the display order information TR of the current picture itself,to thereby correctly decode the timing of refreshment of the framememory. For this reason, even if a picture waiting for refreshment ofthe frame memory is lost, malfunction can be avoided, achieving aneffect of high error resistance.

As still another example, the difference value delta_TR may be encodedfor at least one picture which includes a picture for which the displayorder information TR is larger than the TR_DDR (picture 606 in FIG. 6),and which comes after the random access picture (the most recent DDRpicture herein). Namely, when encoding the display order information ofat least one picture which has display order information TR larger thanTR_DDR (picture 606 in FIG. 6), and which comes after the random accesspicture (the most recent DDR picture herein), the difference valuedelta_TR between the display order information TR of the pertinentpicture and the display order information TR_DDR of the DDR picture maybe encoded, instead of encoding the display order information TR of thepertinent picture itself.

Video Predictive Encoding Program and Video Predictive Decoding Program

The embodiments of the video predictive encoding device can also beimplemented with a video predictive encoding program for controlling acomputer to function as the video predictive encoding device. Likewise,the invention of the video predictive decoding device can also beimplemented with a video predictive decoding program for controlling acomputer to function as the video predictive decoding device.

The video predictive encoding program and the video predictive decodingprogram are provided, for example, as stored in a recording medium, suchas a non-transitory computer readable data storage medium. Examples ofsuch recording media include recording media such as flexible disks,CD-ROMs, and DVDs, or recording media such as ROMs, or semiconductormemories or the like.

FIG. 9 illustrates an example of modules of the video predictiveencoding program for controlling a computer to function as thepreviously discussed video predictive encoding device. As shown in FIG.9, the video predictive encoding program P100 is provided with inputmodule P101, encoding module P102, restoration module P103, picturestorage module P104, and memory management module P105.

In the example of FIG. 9, the input module P101 may include, forexample, at least part of the functionality described with regard to theinput unit 101 and 113. The encoding module P102 may include, forexample, at least part of the functionality described with regard to theblock divider unit 102, the adder unit 105, the transformer 106, thequantizer unit 107, the entropy encoder unit 111, and the predictedsignal generator unit 103. The restoration module P103 may include, forexample, at least part of the functionality described with regard to thede-quantizer unit 108, and the inverse-transformer unit 109. The picturestorage module P104 may include, for example, at least part of thefunctionality described with regard to the frame memory 104. The memorymanagement module P105 may include, for example, at least part of thefunctionality described with regard to the frame memory management unit114. In other examples, more or fewer modules may be used to describethe functionality of the video predictive encoding system. In addition,functionality described as included in an example module may be includedin a different module, or may be divided among multiple modules that areexecutable by a processor included in one or more computers to controlthe functionality of the video predictive encoding device.

The computer may include one or more processors, such as a centralprocessing unit (CPU), one or more digital signal processor, or somecombination of different or the same processors. A processor may be acomponent in a variety of systems. A processor may be one or moregeneral processors, digital signal processors, application specificintegrated circuits, field programmable gate arrays, servers, networks,digital circuits, analog circuits, combinations thereof, or other nowknown or later developed devices for analyzing and processinginstructions and data. A processor may implement a software program,such as code generated manually or programmed into memory, which mayalso include one or more units, or modules. The modules, functions,acts, or tasks illustrated in the figures and/or described herein may beperformed by a programmed processor executing instructions stored in thememory. The modules, functions, acts or tasks may be independent of theparticular type of instructions set, storage media, processor orprocessing strategy and may be performed by software, hardware,integrated circuits, firm-ware, micro-code and the like, operating aloneor in combination. Processing strategies may include multiprocessing,multitasking, parallel processing and the like

FIG. 10 shows modules of the video predictive decoding program forcontrolling a computer to function as the video predictive decodingdevice. As shown in FIG. 10, the video predictive decoding program P200is provided with input module P201, restoration module P202, picturestorage module P203, and memory management module P204 that areexecutable by the processor.

In the example of FIG. 10, the input module P201 may include, forexample, at least part of the functionality described with regard to theinput unit 201. The restoration module P202 may include, for example, atleast part of the functionality described with regard to the dataanalyzer unit 202, the de-quantizer unit 203, the inverse transformerunit 204, the adder unit 205, and the predicted signal generator unit208. The picture storage module P203 may include, for example, at leastpart of the functionality described with regard to the frame memory 207.The memory management module P204 may include, for example, at leastpart of the functionality described with regard to the frame memorymanagement unit 209. In other examples, more or fewer modules may beused to describe the functionality of the video predictive decodingsystem. In addition, functionality described as included in an examplemodule may be included in a different module, or may be divided amongmultiple modules that are executable by a processor included in one ormore computers to control the functionality of the video predictivedecoding device.

The video predictive encoding program P100 and the video predictivedecoding program P200 configured as described above can be stored in arecording medium 10 shown in FIG. 8 and are executed by computer 30described below.

FIG. 7 is a drawing showing a hardware configuration of an examplecomputer for executing a program recorded in a recording medium and FIG.8 is a schematic view of an example computer for executing a programstored in a recording medium. The computer may be, a DVD player, aset-top box, a cell phone, etc. which are provided with at least oneprocessor and are configured to perform processing and control bysoftware, hardware, and some combination of software and hardware.

As shown in the example of FIG. 7, the computer 30 can be provided witha reading device 12 such as a flexible disk drive unit, a CD-ROM driveunit, or a DVD drive unit, a working memory (RAM) 14 in which anoperating system is resident, a memory 16 for storing programs and data,which may also or alternatively be stored elsewhere, such as stored inthe recording medium 10, a monitor unit 18 like a display, a mouse 20and a keyboard 22 as input devices, a communication device 24 fortransmission and reception of data or the like, and a CPU 26 forcontrolling execution of programs. In one example embodiment, when therecording medium 10 is put into the reading device 12, the computer 30becomes accessible to the video predictive encoding program stored inthe recording medium 10, through the reading device 12 and becomes ableto operate as the video predictive encoding device according to thepresent invention, based on the video predictive encoding program.Similarly, in another example, when the recording medium 10 is put intothe reading device 12, the computer 30 becomes accessible to the videopredictive decoding program stored in the recording medium 10, throughthe reading device 12 and becomes able to operate as the videopredictive decoding device according to the present invention, based onthe video predictive decoding program.

As shown in FIG. 8, the video predictive encoding program or the videopredictive decoding program may be provided to the computer in the formof computer data signal 40 superimposed on a carrier wave, through anetwork. In this case, the computer 30 can execute the program after thevideo predictive encoding program or the video predictive decodingprogram received by the communication device 24 is stored into thememory 16, which is a non-transitory computer readable data storagemedium.

LIST OF REFERENCE SIGNS

10: recording medium; 30: computer; 100: video predictive encodingdevice; 101: input terminal; 102: block divider; 103: predicted signalgenerator; 104: frame memory; 105: subtracter; 106: transformer; 107:quantizer; 108: de-quantizer; 109: inverse-transformer; 110: adder; 111:entropy encoder; 112: output terminal; 113: input terminal; 114: framememory management unit; 200: video predictive decoding device; 201:input terminal; 202: data analyzer; 203: de-quantizer; 204:inverse-transformer; 205: adder; 206: output terminal; 207: framememory; 208: predicted signal generator; 209: frame memory managementunit; P100: video predictive encoding program; P101: input module; P102:encoding module; P103: restoration module; P104: picture storage module;P105: memory management module; P200: video predictive decoding program;P201: input module; P202: restoration module; P203: picture storagemodule; P204: memory management module.

The invention claimed is:
 1. A video predictive encoding device comprising: a processor; an input module executed by the processor to accept input of a plurality of pictures constituting a video sequence; an encoding module executed by the processor to encode each of the input pictures by either intra-frame prediction or inter-frame prediction and to generate compressed picture data including a random access picture serving as a picture of random access, the encoding module further executed to encode data providing display order information of each of the pictures; a restoration module executed by the processor to decode the generated compressed picture data to restore a reproduced picture; a picture storage module executed by the processor to store the restored reproduced picture as a reference picture for use in encoding a subsequent picture; and a memory management module executed by the processor to control the picture storage means, wherein after execution of an encoding process by the encoding module to generate the random access picture, the memory management module is further executed by the processor to refresh the picture storage module by setting every reference picture stored in the picture storage module except for the random access picture as unnecessary, the picture storage module refreshed immediately before or immediately after execution by the encoding module to encode a picture having display order information larger than display order information of the random access picture.
 2. The video predictive encoding device according to claim 1, wherein the encoding module is further executed to encode a difference value representing a difference between display order information of at least one encoding target picture and the display order information of the random access picture, the difference value encoded as data providing the display order information of the at least one encoding target picture, including the picture having the display order information larger than the display order information of the random access picture, and which is a first encoding target after completion of the encoding process of generating the random access picture.
 3. The video predictive encoding device according to claim 1, wherein the encoding module is further executed by the processor to encode the display order information of each picture in the video sequence from a picture which is a next encoding target after the random access picture, to the picture having the display order information larger than the display order information of the random access picture, which is a first encoding target after completion of the encoding process of generating the random access picture, the encoding module being further executed to encode as the data providing the display order information of each picture a difference value representing a difference between display order information of each picture and the display order information of the random access picture.
 4. A video predictive decoding device comprising: a processor; an input module executed by the processor to accept input of compressed picture data including a random access picture serving as a picture of random access, the compressed picture data obtained by encoding each of a plurality of pictures constituting a video sequence by either intra-frame prediction or inter-frame prediction, and the input module further executed to accept input of display order encoded data obtained by encoding data providing display order information of each of the pictures; a restoration module executed by the processor to decode the compressed picture data to restore a reproduced picture, the restoration module further executed to decode the display order encoded data to restore the display order information; a picture storage module executed by the processor to store the restored reproduced picture as a reference picture for use by the restoration module to decode a subsequent picture; and a memory management module executed by the processor to control the picture storage module, wherein after completion of a decoding process by the restoration module to decode the random access picture, the memory management module is further executed to refresh the picture storage module by setting every reference picture stored in the picture storage module, except for the random access picture, as unnecessary, the picture storage module refreshed immediately before or immediately after the restoration module is executed to decode a picture which has display order information larger than the display order information of the random access picture.
 5. The video predictive decoding device according to claim 4, wherein the restoration module is further executed to decode display order information of at least one decoding target picture, which includes decoding the picture having display order information larger than the display order information of the random access picture and which is a first decoding target after completion of the decoding process of decoding the random access picture, the restoration module is further executed to restore the display order information of the decoding target picture by addition of a difference value to the display order information of the random access picture, the difference value representing a difference between the display order information of the decoding target picture and the display order information of the random access picture, the difference value obtained by decoding the display order encoded data of the decoding target picture.
 6. The video predictive decoding device according to claim 4, wherein display order information of each picture is decoded in a sequence from a picture that is a next decoding target after the random access picture, to the picture having display order information larger than the display order information of the random access picture and which is a first decoding target after completion of a decoding process of generating the random access picture, and wherein the restoration module is further executed to restore display order information of each picture by addition of a difference value to the display order information of the random access picture, the difference value representing a difference between the display order information of each picture and the display order information of the random access picture, the difference value obtained by decoding the display order encoded data of each picture.
 7. A video predictive encoding method executed by a video predictive encoding device with a picture storage medium for storing a reference picture used for encoding of a subsequent picture, the method comprising: an input step of a processor accepting input of a plurality of pictures constituting a video sequence; an encoding step of the processor encoding each of the pictures by either intra-frame prediction or inter-frame prediction to generate compressed picture data including a random access picture serving as a picture of random access, and encoding data providing display order information of each of the pictures; a restoration step of the processor decoding the generated compressed picture data to restore a reproduced picture; a picture storage step of storing in the picture storage medium the restored reproduced picture as the reference picture used for encoding of the subsequent picture; and a memory management step of the processor controlling the picture storage medium, wherein after completion of an encoding process of generating the random access picture, in the memory management step, the processor refreshing the picture storage medium by setting every reference picture stored in the picture storage medium, except for the random access picture, as unnecessary, the picture storage medium refreshed immediately before or immediately after encoding a picture with display order information larger than the display order information of the random access picture.
 8. The video predictive encoding method according to claim 7, wherein in the encoding step, the processor encoding a difference value between display order information of at least one encoding target picture and the display order information of the random access picture, the difference value encoded as data providing the display order information of at least one encoding target picture including a picture having display order information larger than the display order information of the random access picture and which is a first encoding target after completion of the encoding process of generating the random access picture.
 9. The video predictive encoding method according to claim 7, wherein in the encoding step, for each picture in a sequence from a picture that is a next encoding target after the random access picture, to a picture having display order information larger than the display order information of the random access picture and which is a first encoding target after completion of the encoding process of generating the random access picture, the processor further encoding, as the data providing display order information of each picture, a difference value between the display order information of each picture and the display order information of the random access picture.
 10. A video predictive decoding method executed by a video predictive decoding device with a picture storage medium for storing a reference picture used for decoding of a subsequent picture, the method comprising: an input step of a processor accepting input of compressed picture data including a random access picture serving as a picture of random access, the compressed picture data obtained by encoding each of a plurality of pictures constituting a video sequence by either intra-frame prediction or inter-frame prediction, and the input step also accepting input of display order encoded data obtained by encoding data providing display order information of each of the pictures; a restoration step of the processor decoding the compressed picture data to restore a reproduced picture and decoding the display order encoded data to restore the display order information; a picture storage step of the processor storing in the picture storage medium the restored reproduced picture as the reference picture to be used for decoding of the subsequent picture; and a memory management step of the processor controlling the picture storage medium, wherein following completion of a decoding process of decoding the random access picture, in the memory management step, the processor refreshing the picture storage means by setting every reference picture stored in the picture storage means, except for the random access picture, as unnecessary immediately before or immediately after decoding a picture having display order information larger than the display order information of the random access picture.
 11. The video predictive decoding method according to claim 10, wherein in the restoration step, when the processor is decoding display order information of at least one decoding target picture that includes decoding the picture having display order information larger than the display order information of the random access picture, and which is the first decoding target after completion of the decoding process of decoding the random access picture, the processor restoring the display order information of the decoding target picture by adding a difference value to the display order information of the random access picture, the difference value representing a difference between the display order information of the decoding target picture and the display order information of the random access picture, the difference value obtained by the processor decoding the display order encoded data of the decoding target picture.
 12. The video predictive decoding method according to claim 10, wherein in the restoration step, when the processor is decoding display order information of each picture in a sequence from a picture which is a next decoding target after the random access picture, to a picture having display order information larger than the display order information of the random access picture, and which is the first decoding target after completion of a decoding process of generating the random access picture, the processor restoring the display order information of each picture by adding a difference value to the display order information of the random access picture, the difference value representing a difference between the display order information of each picture and the display order information of the random access picture, the difference value obtained by decoding the display order encoded data of each picture.
 13. A computer readable non-transitory storage medium storing a video predictive encoding program having a plurality of instructions executable by a computer, the computer readable storage medium comprising: instructions executable by the computer to accept input of a plurality of pictures constituting a video sequence; instructions executable by the computer to encode each of the input pictures by either intra-frame prediction or inter-frame prediction to generate compressed picture data, the compressed picture data including a random access picture serving as a picture of random access, and instructions executed by the computer to encode data providing display order information of each of the pictures; instructions executable by the computer to decode the generated compressed picture data, to restore a reproduced picture; instructions executable by the computer to store the restored reproduced picture as a reference picture for use in encoding of a subsequent picture; and instructions executable by the computer, following completion of an encoding process to generate the random access picture, to refresh a picture storage medium by setting every reference picture stored in the picture storage medium, except for the random access picture, as unnecessary, the picture storage medium refreshed immediately before or immediately after encoding a picture having display order information larger than the display order information of the random access picture.
 14. A computer readable non-transitory storage medium storing a video predictive decoding program having a plurality of instructions executable by a computer, the computer readable storage medium comprising: instructions executable by the computer to accept input of compressed picture data including a random access picture serving as a picture of random access, which was obtained by encoding each of a plurality of pictures constituting a video sequence, by either intra-frame prediction or inter-frame prediction, and to accept input of display order encoded data obtained by encoding data providing display order information of each of the pictures; instructions executable by the computer to decode the compressed picture data to restore a reproduced picture, and to decode the display order encoded data to restore the display order information; instructions executable by the computer to store the restored reproduced picture, as a reference picture used to decode a subsequent picture; and instructions executable by the computer following completion of a decoding process of decoding the random access picture, to refresh the picture storage medium by setting every reference picture stored in the picture storage medium, except for the random access picture, as unnecessary, the picture storage medium refreshed immediately before or immediately after decoding a picture having display order information larger than the display order information of the random access picture. 